Introduction
The development of modern unmanned systems places higher demands on the performance of the controllers. Typically, these systems are constrained, and violation of constraints may lead to unsafe behaviors that can severely affect system operation. Model predictive control (MPC) is capable of providing an optimal solution while handling the constraints explicitly, which has seen significant success in recent decades with wide applications in diverse fields [1], [2], [3], [4], [5], [6].
The successful applications of MPC attracted tremendous academic interest, and soon a rigorous stability-centered MPC theoretical foundation was established. The early stability results on MPC employ a zero-state terminal equality constraint [7], [8]. They were subsequently extended to the use of a terminal inequality constraint by taking a control Lyapunov function as the terminal cost [9], [10], [11], which established the currently well-known framework for stability. Based on this framework, a commonly adopted MPC design paradigm involves finding an appropriate terminal cost and a terminal controller as well as a terminal set that strictly meet specific conditions. While this is relatively easy for linear systems, it is generally a challenging task for nonlinear ones. Even if a qualified terminal cost, a terminal controller, and the corresponding terminal set are found, the conditions for stability and recursive feasibility are usually quite conservative. In practice, however, the theoretically well-established MPC approaches are prohibitively difficult to apply due to the high computational burden. Instead, MPC without terminal cost and terminal constraint enjoys wide popularity. Its closed-loop stability and recursive feasibility can be guaranteed under some conditions [12], [13]. But the absence of terminal cost entails that the residual costs outside the truncated prediction horizon are completely ignored, which makes it difficult to achieve optimal performance under a very limited prediction horizon. Motivated by the analysis above, we aim to develop a learning-based MPC scheme to learn an appropriate terminal cost function without complex offline design procedures while improving the performance of the controller.
In recent years, striking progress of reinforcement learning (RL), such as AlphaGo [14] and Atari games [15], has drawn the attention of the control community. Different from MPC’s offline design, RL optimizes the control policy through online data-based adaptation [16], [17]. Observed state transitions and costs (or rewards) are the only inputs to the RL agent, and no prior knowledge of the system dynamics is needed [18], [19]. A central idea in RL is temporal-difference (TD) learning [20], [21], [22], which estimates the value function directly from raw experience in a bootstrapping way, without waiting for a final outcome. The value function is a prediction of the expected long-term reward at each state. When the optimality is reached, it encodes the global optimal information so that the infinite-horizon optimal control policy can be obtained.
Nevertheless, to approximate the global optimal policy, the RL agent tends to try different policies and learns via trial and error, thereby struggling to provide safe guarantees on the resulting behaviors. This problem becomes prominent when confronted with some safety-critical systems, and safety in this context is defined in terms of stability. There have been some achievements centered on safe RL recently [23], [24], [25], but it remained largely an open field.
In view of the powerful data-driven optimization capability of RL, if combined with MPC to optimize the controller design with running data, it can be expected to improve the control performance. At the same time, benefiting from the solid theoretical foundation of MPC, the properties of the behaviors produced by the RL agent would be easier to analyze. This motivates the attempts to integrate the best of both worlds. However, only limited pioneering work has been reported in this field. For example, in [26], a novel “plan online and learn offline” framework was proposed by taking MPC as the trajectory optimizer of RL. It is shown that MPC enables a more efficient exploration, thereby accelerating the convergence and reducing the approximation errors in the value function. The related work was extended to the framework of actor-critic (AC) in [27] to handle the case of sparse and binary reward, which shows that MPC is an important sampler that effectively propagates global information. These results were further applied to the tasks defined by simple and easily interpretable high-level objectives on a real unmanned ground vehicle in [28], and the expression of the value function was improved. Yet, the focus of these works is on the improvements that MPC brings to RL practice, with no reference to stability. The stability issue was highlighted in [29] where an economic MPC was used as a function approximator in RL. This scheme was further analyzed in [30] under an improved algorithmic framework. These two papers paved the way for [31], where the constraint satisfaction concern was addressed with the help of robust linear MPC. The practicality of these approaches, however, remains to be investigated and verified. To the best of the authors’ knowledge, few studies can balance the theoretical and practical guarantees of the “RL + MPC” approaches, that is, to provide a theoretically safe and operationally efficient scheme, and this article aims to fill this gap.
We propose an “RL + MPC” scheme from a new perspective: policy iteration (PI). Considering that MPC’s ability to handle constraints can provide constraint-enforcing policies for RL agents, safety referred to in this article is not only limited to stability, but also constraint satisfaction. The main contributions of this article contain threefold as follows.
We provide a new idea for combining RL and MPC by bridging them through PI. In this way, a complete RL-based model predictive control (RLMPC) scheme is developed for discrete-time systems. In each iteration, the current policy is evaluated through learning to obtain the value function. Then it is employed as the terminal cost to compensate for the suboptimality induced by the truncated prediction horizon of MPC, thus improving the policy. This solves the challenge of complex offline design procedures while progressively improving the performance to (near-)optimum.
The convergence of learning, recursive feasibility, and closed-loop stability of the proposed RLMPC are closely investigated, thereby theoretically guaranteeing its safety. We demonstrate that no constraint is violated even before the optimal value function is well-learned, which verifies the ability of MPC to effectively constrain the RL-produced policies within safe limits.
We incorporate the value function approximation (VFA) technique into the developed RLMPC approach to approximate the global optimal value function with high computational efficiency. The effect of the prediction horizon on the control performance is scrutinized. Results show that the proposed RLMPC scheme can achieve nearly the same optimal performance as traditional MPC in linear system control and outperform it in nonlinear cases, which is due to the conservativeness of the offline MPC design for nonlinear systems. In addition, thanks to the elimination of the terminal controller and terminal constraint, the RLMPC scheme is particularly useful in dealing with systems where it is difficult or even impossible to design a terminal controller, such as nonholonomic systems, while reducing computational burden by flexibly adjusting its prediction horizon.
The rest of this article is organized as follows. In Section II, we formally outline the problem formulation. The RLMPC scheme is developed in Section III. Its convergence, optimality, feasibility, and stability properties are analyzed in Section IV. Section V discusses the implementation of the overall scheme. We test the proposed scheme in simulations of both linear and nonlinear examples in Section VI. Section VII provides final conclusion.
Notation: We use the following convention:
Problem Formulation
A. Optimal Control Problem
Consider a dynamic system described by the following state space difference equation:
![Right-click on figure for MathML and additional features.](data:image/svg+xml,<svg xmlns="http://www.w3.org/2000/svg" width="24" height="20"><rect fill-opacity="0"/></svg> "Right-click on figure or equation for MathML and additional features."){kind=small}
\begin{equation*} x_{k+1} = f(x_{k},u_{k}) \tag{1}\end{equation*}
where \begin{equation*} x_{k} \in \mathbb {X}, \quad u_{k} \in \mathbb {U} \tag{2}\end{equation*}
Assumption 1:
The function
Remark 1:
Assumption 1 is a nonintrusive and common assumption in the MPC community and can be found in numerous literatures (e.g., [9], [10], [11], [12], [13]). For a system
For the particular setup considered above, the infinite-horizon optimal control problem at time
Problem 1:
Find a control policy
\begin{equation*} J_{\infty }(x_{k}, \boldsymbol {u}_{\infty}) = \sum _{i=k}^{\infty } \ell (x_{i}, u_{i}) \tag{3}\end{equation*}
subject to constraints (1) and (2), with the running cost \begin{align*} \ell (0, 0)=&0 \\ \alpha _{1}^{\mathcal {K}_{\infty} }(\|x \|)\leq&\check {\ell }(x) \triangleq \inf _{u \in \mathbb {U}} \ell (x,u) \leq \alpha _{2}^{\mathcal {K}_{\infty }}(\|x \|) \tag{4}\end{align*}
Remark 2:
The running cost defined by (4) is a fairly general form and is common in many MPC research studies (e.g., [13], [33]). Its specific form is task-dependent, for example, it usually takes a quadratic form in regulation problems:
B. Background of Model Predictive Control
Generally, there is no analytic solution to Problem 1, especially for constrained nonlinear systems. An efficient method to solve this problem is MPC [9], which provides an approximated solution by recursively solving the following OP 1 at each
OP 1: Find the optimal control sequence
\begin{align*} &\!\min _{ \boldsymbol {u}_{k}}\, J(x_{k}, \boldsymbol {u}_{k})=\sum _{i=0}^{N-1} \ell (x_{i|k}, u_{i|k}) + F(x_{N|k}) \tag{5a}\\ &\text {s.t.} x_{0|k} = x_{k} \tag{5b}\\ &\hphantom {\text {s.t.} } x_{i+1|k} = f(x_{i|k},u_{i|k}),\quad 0\le i\le N - 1 \tag{5c}\\ &\hphantom {\text {s.t.} } u_{i|k} \in \mathbb {U}, \quad 0\le i\le N-1 \tag{5d}\\ &\hphantom {\text {s.t.} } x_{i|k}\in \mathbb {X}, \quad 1\le i\le N-1 \tag{5e}\\ &\hphantom {\text {s.t.} } x_{N|k} \in \mathbb {X}_{\Omega} \tag{5f}\end{align*}
Where After solving OP 1, apply only the first element in
To guarantee the recursive feasibility and stability, it is required that
As mentioned in Section I, it is generally a difficult task to design such terminal conditions (i.e., terminal cost, terminal constraint, and terminal controller) for nonlinear systems. Although MPC without terminal condition (MPCWTC) is proposed as an option to circumvent this challenge, its performance under a very limited (but necessary) prediction horizon tends to be inferior to that of the kind with terminal cost. Therefore, we propose to introduce RL into MPC to learn the terminal cost online, thereby generating a policy that is closer to the optimal one in the sense of an infinite horizon.
C. Background of RL
An important class of RL techniques aims to learn the optimal policy
\begin{equation*} V_{\pi} (x_{k}) = \ell (x_{k}, u_{k}) + V_{\pi} (x_{k+1}). \tag{6}\end{equation*}
According to Bellman’s principle, when the optimality is achieved, the optimal value function
\begin{equation*} V_{\pi }^{\ast} \left ({x_{k}}\right) = \min _{u_{k}} \left \{{ \ell \left ({x_{k}, u_{k}}\right) + V_{\pi }^{\ast}\left ({x_{k+1}}\right) }\right \} \tag{7}\end{equation*}
and the corresponding optimal policy is given by \begin{equation*} \pi ^{\ast} \left ({x_{k}}\right) = \arg \min _{u_{k}} \left \{{\ell \left ({x_{k}, u_{k}}\right) + V_{\pi }^{\ast}\left ({x_{k+1}}\right)}\right \}. \tag{8}\end{equation*}
Nevertheless, the calculation of
\begin{equation*} V_{\pi} (x_{k}) \leftarrow V_{\pi} (x_{k}) + \alpha \left [{V_{\mathrm{ TD}}(x_{k}) - V_{\pi} (x_{k}) }\right] \tag{9}\end{equation*}
where \begin{equation*} V_{\mathrm{ TD}}(x_{k}) = \sum _{i = k}^{k+N-1} \ell (x_{i}, u_{i}) + V_{\pi} (x_{k+N}). \tag{10}\end{equation*}
It can be seen that this TD target has a similar form as the MPC cost function (5a), which builds up the connection between MPC and RL, and their combination gives rise to the RLMPC algorithm to be presented in Section III– V.
RL-Based MPC
In the proposed RLMPC, the value function and control policy are updated in the PI manner. Specifically, we take a currently obtained heuristic term as the terminal cost of MPC, and the corresponding control input is the current control policy. The costs incurred by following this policy, which can be regarded as the rewards for the controller interacting with the system dynamics, are used as training data for the learning of the value function. VFA technique is employed to approximate the underlying value function, that is, the evaluation of the current policy. With the latest learned value function, we take it as the heuristic term in the cost function, thus updating the MPC controller, that is, the current policy. Fig. 1 illustrates the RLMPC pipeline.
A. Policy Generator—MPC
In this section, we formulate the MPC problem to be solved at each step (see OP 2). Since MPC only looks
OP 2: Find the optimizer
\begin{align*} &\!\min _{ \boldsymbol {u}_{k}}\, J(x_{k}, \boldsymbol {u}_{k})= \sum _{i=0}^{N-1} \ell (x_{i|k}, u_{i|k}) + V_{\pi} (x_{N|k}) \tag{11a}\\ &\text {s.t.} x_{0|k} = x_{k} \tag{11b}\\ &\hphantom {\text {s.t.} } x_{i+1|k} = f(x_{i|k},u_{i|k}),0\le i\le N - 1 \tag{11c}\\ &\hphantom {\text {s.t.} } u_{i|k} \in \mathbb {U},\quad 0\le i\le N-1 \tag{11d}\\ &\hphantom {\text {s.t.} } x_{i|k}\in \mathbb {X},\quad 1\le i\le N \tag{11e}\end{align*}
where the control sequence Up to this point, we obtain the policy
Remark 3:
A significant advantage of this formulation lies in the fact that it not only inherits the advantages of MPCWTC [13], but also has the potential to further improve the control performance by compensating the costs neglected by the truncated prediction horizon through the learned heuristic function term.
In this article, the initial policy
Assumption 2 (Controllability Assumption):
There exists a positive constant
\begin{equation*} V_{\pi }^{\ast} (x_{k}) \leq \beta \cdot \check {\ell }(x_{k})\quad \forall x_{k} \in \mathbb {X} \tag{12}\end{equation*}
where Lemma 1 (See[13]):
If the prediction horizon
\begin{equation*} N > \frac { 2\ln \beta }{\ln \beta - \ln (\beta -1)} \tag{13}\end{equation*}
it is guaranteed that the closed-loop system is asymptotically stable and the MPCWTC optimization problem is recursively feasible.Assumption 3:
In this context, the prediction horizon of the initial controller satisfies (13).
Remark 4:
Assumption 2 is a fairly standard condition in the MPCWTC literature (e.g., [13], [37], [38]) and is also known as the boundedness condition of the optimal value function. It indicates the controllability of system (1) with respect to the equilibrium, since otherwise
B. Learning of the Value Function
Now we focus on how to perform policy evaluation on a fixed control policy
According to the higher-order Weierstrass approximation theorem [41], there exists a dense polynomial basis set
\begin{align*} V_{\pi} (x_{k})=&\sum _{i=1}^{\infty } W_{i} \Phi _{i}(x_{k}) = \sum _{i=1}^{p} W_{i} \Phi _{i}(x_{k}) + \sum _{i=p+1}^{\infty } W_{i} \Phi _{i}(x_{k}) \\=&W^{\mathrm {T}} \Phi (x_{k}) + e(x_{k}) = \hat {V}_{\pi} (x_{k}, W) + e(x_{k}) \tag{14}\end{align*}
where Consider a set containing
With all these costs
\begin{align*} \hat {V}_{\pi} (x_{k_{j}}, W) \leftarrow \hat {V}_{\pi} (x_{k_{j}}, W) + \alpha \left [{J\left ({x_{k_{j}}, \boldsymbol {u}^{\ast}_{k_{j}}}\right) - \hat {V}_{\pi} (x_{k_{j}}, W) }\right]\!\!\! \tag{15}\end{align*}
for \begin{align*} W\leftarrow&W-\frac {1}{2}\alpha \nabla _{W} \left [{J\left ({x_{k_{j}}, \boldsymbol {u}^{\ast}_{k_{j}}}\right)-\hat {V}_{\pi} \left ({x_{k_{j}}, W}\right)}\right]^{2} \\=&W+\alpha \left [{J\left ({x_{k_{j}}, \boldsymbol {u}^{\ast}_{k_{j}}}\right)-\hat {V}_{\pi} \left ({x_{k_{j}}, W}\right)}\right]\nabla _{W} \hat {V}_{\pi} \left ({x_{k_{j}}, W}\right) \\=&W+\alpha \left [{J\left ({x_{k_{j}}, \boldsymbol {u}^{\ast}_{k_{j}}}\right)- W^{\mathrm {T}} \Phi \left ({x_{k_{j}}}\right)}\right] \Phi \left ({x_{k_{j}}}\right), \quad j \in \mathbb {N}_{\left [{1,q}\right]} \tag{16}\end{align*}
After
C. PI in RLMPC
In RLMPC, the value function and the control policy are updated by iterations with index
Initialization: Given an initial state
xk∈X k V0π(⋅)≡0 π0(xk) t=0,1,2,… Policy Evaluation Step: Determine the value function
Vt+1π(⋅) πt(⋅) X Sk Jt(xkj,utkj) xkj∈Sk View SourceJt(xkj,utkj)==minukj∈U{∑i=0N−1ℓ(xi|kj,ui|kj)+Vtπ(xN|kj)}∑i=0N−1ℓ(xti|kj,uti|kj)+Vtπ(xtN|kj)(17) \begin{align*} J^{t}(x_{k_{j}}, \boldsymbol {u}_{k_{j}}^{t})=&\min _{ \boldsymbol {u}_{k_{j}} \in \mathbb {U}}\left \{{ \sum _{i=0}^{N-1} \ell (x_{i|k_{j}}, u_{i|k_{j}}) + V^{t}_{\pi }(x_{N|k_{j}}) }\right \} \\=&\sum _{i=0}^{N-1} \ell \left ({x_{i|k_{j}}^{t}, u_{i|k_{j}}^{t}}\right) + V_{\pi }^{t}\left ({x_{N|k_{j}}^{t}}\right) \tag{17}\end{align*}
where xkj=xt0∣kj xti∣kj uti∣kj i∈N[0,N−1] Vt+1π(xk)→Jt(xk,utk)∀xk∈X.(18) \begin{equation*} V^{t+1}_{\pi} (x_{k}) \rightarrow J^{t}(x_{k}, \boldsymbol {u}_{k}^{t})\quad \forall x_{k} \in \mathbb {X}. \tag{18}\end{equation*}
Policy Improvement Step: According to the descriptions in Section III-A, the updated policy
πt+1(xk) Vt+1π(⋅) View Sourceut+1k=argminut+1k∈U{∑i=0N−1ℓ(xi|k,ui|k)+Vt+1π(xN|k)}(19) \begin{equation*} \boldsymbol {u}_{k}^{t+1}=\arg \min _{ \boldsymbol {u}^{t+1}_{k} \in \mathbb {U}} \left \{{\sum _{i=0}^{N-1} \ell (x_{i|k}, u_{i|k}) + V_{\pi }^{t+1}(x_{N|k})}\right \}\quad \tag{19}\end{equation*}
and ut+1k
In what follows, we focus on analyzing the theoretical properties of this RLMPC scheme.
Convergence, Feasibility, and Stability Analysis
In this section, we analyze the convergence, feasibility, and stability properties. We show that for
Definition 1:
For the RLMPC optimization problem OP 2, a control sequence
A. Convergence Analysis
We first present two theorems to respectively demonstrate that the value function
Theorem 1:
For
Proof:
Define a new value function
\begin{equation*} \Gamma ^{t}(x_{k})= \sum _{i=0}^{N-1} \ell (\tilde {x}_{i|k}, \tilde {u}_{i|k}) + \Gamma ^{t-1}(\tilde {x}_{N|k}) \tag{20}\end{equation*}
where \begin{align*} \Gamma ^{1}(x_{k})=&\sum _{i=0}^{N-1} \ell (\tilde {x}_{i|k}, \tilde {u}_{i|k}) + \Gamma ^{0}(\tilde {x}_{N|k}) \\\geq&\min _{ \boldsymbol {u}_{k} \in \mathbb {U}}\left \{{ \sum _{i=0}^{N-1} \ell (x_{i|k}, u_{i|k}) + \Gamma ^{0}(x_{N|k}) }\right \} \\=&\min _{ \boldsymbol {u}_{k} \in \mathbb {U}}\left \{{ \sum _{i=0}^{N-1} \ell (x_{i|k}, u_{i|k}) + V^{0}_{\pi }(x_{N|k}) }\right \} \\=&\sum _{i=0}^{N-1} \ell \left ({x_{i|k}^{0}, u^{0}_{i|k}}\right) + V^{0}_{\pi }\left ({x_{N|k}^{0}}\right) =V^{1}_{\pi }\left ({x_{k}}\right). \tag{21}\end{align*}
The last two equations hold from (17) and (18), indicating the following update law:
\begin{align*} V_{\pi } ^{t+1} \left ({x_{k}}\right)=&J^{t}\left ({x_{k}, \boldsymbol {u}_{k}^{t}}\right) \\=&\sum _{i=0}^{N-1} \ell \left ({x_{i|k}^{t}, u_{i|k}^{t}}\right) + V_{\pi }^{t}\left ({x_{N|k}^{t}}\right) \\=&\min _{ \boldsymbol {u}_{k} \in \mathbb {U}} \left \{{\sum _{i=0}^{N-1} \ell (x_{i|k}, u_{i|k}) + V_{\pi }^{t}(x_{N|k})}\right \}. \tag{22}\end{align*}
Assume that at
\begin{align*} \Gamma ^{l+1}(x_{k})=&\sum _{i=0}^{N-1} \ell (\tilde {x}_{i|k}, \tilde {u}_{i|k}) + \Gamma ^{l}(\tilde {x}_{N|k})\\\geq&\min _{ \boldsymbol {u}_{k} \in \mathbb {U}}\left \{{ \sum _{i=0}^{N-1} \ell (x_{i|k}, u_{i|k}) + \Gamma ^{l}(x_{N|k}) }\right \}\\=&\sum _{i=0}^{N-1} \ell \left ({\bar {x}^{l}_{i|k}, \bar {u}^{l}_{i|k}}\right) + \Gamma ^{l}\left ({\bar {x}^{l}_{N|k}}\right) \\\geq&\sum _{i=0}^{N-1} \ell \left ({\bar {x}^{l}_{i|k}, \bar {u}^{l}_{i|k}}\right) + V_{\pi }^{l}\left ({\bar {x}^{l}_{N|k}}\right) \\\geq&\min _{ \boldsymbol {u}_{k} \in \mathbb {U}}\left \{{ \sum _{i=0}^{N-1} \ell (x_{i|k}, u_{i|k}) + V^{l}_{\pi }(x_{N|k}) }\right \}\\=&\sum _{i=0}^{N-1} \ell \left ({x_{i|k}^{l}, u^{l}_{i|k}}\right) + V^{l}_{\pi }\left ({x^{l}_{N|k}}\right) =V^{l+1}_{\pi }\left ({x_{k}}\right)\end{align*}
where \begin{equation*} \forall t: \Gamma ^{t}(x_{k}) \geq V^{t}_{\pi }(x_{k}). \tag{23}\end{equation*}
Next, we use this conclusion to prove that
\begin{equation*} \Gamma ^{l}\left ({x_{k}}\right) = \sum _{i=0}^{N-1} \ell \left ({x^{l}_{i|k}, u_{i|k}^{l}}\right) + \Gamma ^{l-1}\left ({x^{l}_{N|k}}\right). \tag{24}\end{equation*}
At this time, assume that
\begin{equation*} \Gamma ^{l}\left ({x_{k}}\right)-V_{\pi }^{l+1}\left ({x_{k}}\right) = \Gamma ^{l-1}\left ({x^{l}_{N|k}}\right)-V_{\pi }^{l}\left ({x^{l}_{N|k}}\right) \leq 0. \tag{25}\end{equation*}
Hence,
\begin{equation*} \forall t: V^{t}_{\pi }(x_{k}) \leq \Gamma ^{t}(x_{k}) \leq V^{t+1}_{\pi }(x_{k}) \tag{26}\end{equation*}
which completes the proof.Remark 5:
Since
Theorem 2:
For
\begin{equation*} V^{\ast}_{\pi }(x_{k}) = \min _{ \boldsymbol {u}_{k} \in \mathbb {U}} \left \{{\sum _{i=0}^{N-1} \ell (x_{i|k}, u_{i|k}) + V_{\pi }^{\ast}(x_{N|k})}\right \} \tag{27}\end{equation*}
then \begin{equation*} \forall t: V^{t}_{\pi }(x_{k}) \leq V^{\ast}_{\pi }(x_{k}). \tag{28}\end{equation*}
Proof:
We prove this by induction. First, at
\begin{align*} V_{\pi} ^{l+1}(x_{k})=&\min _{ \boldsymbol {u}_{k} \in \mathbb {U}} \left \{{\sum _{i=0}^{N-1} \ell (x_{i|k}, u_{i|k}) + V_{\pi }^{l}(x_{N|k})}\right \} \\\leq&\min _{ \boldsymbol {u}_{k} \in \mathbb {U}} \left \{{\sum _{i=0}^{N-1} \ell (x_{i|k}, u_{i|k}) + V_{\pi }^{\ast}(x_{N|k})}\right \} \\=&V^{\ast}_{\pi }(x_{k}). \tag{29}\end{align*}
Thus,
Based on the properties revealed by Theorems 1 and 2, we give the following theorem to show the convergence of RLMPC.
Theorem 3:
For
Proof:
From Theorems 1 and 2, we know that
\begin{align*} V_{\pi} ^{\infty }(x_{k})=&\lim _{t \rightarrow \infty } V_{\pi} ^{t+1}(x_{k}) \\=&\lim _{t \rightarrow \infty } \min _{ \boldsymbol {u}_{k} \in \mathbb {U}} \left \{{\sum _{i=0}^{N-1} \ell (x_{i|k}, u_{i|k}) + V_{\pi }^{t}(x_{N|k})}\right \} \\=&\min _{ \boldsymbol {u}_{k} \in \mathbb {U}} \left \{{\sum _{i=0}^{N-1} \ell (x_{i|k}, u_{i|k}) + \lim _{t \rightarrow \infty } V_{\pi }^{t}(x_{N|k})}\right \} \\=&\min _{ \boldsymbol {u}_{k} \in \mathbb {U}} \left \{{\sum _{i=0}^{N-1} \ell (x_{i|k}, u_{i|k}) + \lim _{t \rightarrow \infty } V_{\pi }^{t+1}(x_{N|k})}\right \} \\=&\sum _{i=0}^{N-1} \ell \left ({x^{\infty }_{i|k}, u^{\infty }_{i|k}}\right) + V_{\pi} ^{\infty }\left ({x^{\infty }_{N|k}}\right)\end{align*}
which is the Bellman optimality equation. Thus, B. Stability and Feasibility Analysis
Theorem 4:
For
Proof:
For
\begin{equation*} V_{\tilde {N}}(x_{k})=\min _{ \boldsymbol {u}_{\tilde {N}\parallel k} \in \mathbb {U}} \sum _{i=0}^{\tilde {N}-1} \ell (x_{i|k}, u_{i|k}) \tag{30}\end{equation*}
subject to (11b)–(11e) with \begin{align*} V_{2N}(x_{k})=&\min _{ \boldsymbol {u}_{2N \parallel k} \in \mathbb {U}} \sum _{i=0}^{2N-1} \ell (x_{i|k}, u_{i|k}) \\=&\min _{ \boldsymbol {u}_{2N \parallel k} \in \mathbb {U}} \left \{{ \sum _{i=0}^{N-1} \ell (x_{i|k}, u_{i|k}) + \sum _{i=N}^{2N-1} \ell (x_{i|k}, u_{i|k}) }\right \} \\=&\min _{ \boldsymbol {u}_{N \parallel k} \in \mathbb {U}} \left \{{ \sum _{i=0}^{N-1} \ell (x_{i|k}, u_{i|k}) + V_{N}(x_{N|k}) }\right \} \tag{31}\end{align*}
\begin{align*} V_{2N}(x_{k})=&\min _{ \boldsymbol {u}_{N \parallel k} \in \mathbb {U}} \left \{{ \sum _{i=0}^{N-1} \ell (x_{i|k}, u_{i|k}) + V_{\pi }^{1}(x_{N|k}) }\right \} \\=&J^{1}(x_{k}, \boldsymbol {u}_{N \parallel k}^{1})=V_{\pi }^{2}(x_{k}). \tag{32}\end{align*}
This illustrates that the policy generated by RLMPC with
\begin{equation*} V_{(t+1) \cdot N}(x_{k})=J^{t}(x_{k}, \boldsymbol {u}_{N \parallel k}^{t})=V_{\pi }^{t+1}(x_{k}) \quad \forall t \in \mathbb {N} \tag{33}\end{equation*}
which implies that the policy Remark 6:
The above proof reveals that the essence of RLMPC is to achieve the effect of accumulating the prediction horizon on the basis of the initial controller through iterations. As
Remark 7:
For specific control tasks, the MPCWTC could generally achieve (near-)optimal performance with a very limited prediction horizon
Implementation Issues on RLMPC
A. On the PI
Theorems 1 and 2 reveal that as
As we can see in Section III-B, the update of
Remark 8:
In contrast to PI, value iteration (VI) is considered less computationally expensive by reducing the policy evaluation step to a single iteration. Although it is possible to adapt RLMPC to the VI update manner, the properties presented in Section IV may be lost. This is due to the fact that the intermediate policies of VI are not guaranteed to be stable and the intermediate value functions may not correspond to any policy. Taking such value functions as the terminal costs, it is difficult to ensure that the corresponding RLMPC generates suitable policies for value function updates. How to ensure the convergence, stability, and feasibility in the case of VI would be our future research.
B. Overall Algorithm of the RLMPC Scheme
Given the state
The pseudo-code of the RLMPC algorithm is shown in Algorithm 1, which is an example of updating the policy at every control step, that is,
Algorithm 1 RLMPC Algorithm
Initialize:
for
Measure the current state
Solve OP 2 for
Apply
if
Construct sample set
for
Solve OP 2 for
Calculate
Update
if
end if
end for
end if
if
end if
end for
Simulation Examples
To evaluate the effectiveness of the proposed RLMPC scheme, two examples are studied: one involving a linear system and the other involving a nonlinear one. As is known, traditional MPC (which refers to the one described in OP 1) can attain linear quadratic optimal performance for linear systems with properly designed terminal conditions, so the linear system example can be a benchmark for RLMPC performance. For nonlinear systems, however, it can only achieve suboptimal control performance, which is where RLMPC comes in. We will highlight the superiority of RLMPC in the nonlinear system example. In both examples, we investigate RLMPC (following Algorithm 1) for two episodes. The first episode (labeled as “RLMPC Epi. 1”), called the learning episode, is to learn from scratch, starting with the initial policy to control the system; while the second episode (labeled as “RLMPC Epi. 2”), also referred to as the re-execution episode, is to redo the task under the learned value function (LVF) to check the performance.
A. Linear System
Consider the following linear system:
\begin{equation*} x_{k+1} = A x_{k} + B u_{k} \tag{34}\end{equation*}
where The relevant state trajectories and control inputs are reported in Fig. 2. It can be seen that the trajectories of “RLMPC Epi. 1” and “MPCWTC” are very close in the initial few steps, which is due to the fact that the initial policy of the learning episode is essentially the MPCWTC. As the policy of the learning episode improves with each control step
Fig. 4 illustrates the accumulated costs (ACCs), which is defined as the sum of the running costs from the initial time to the current time
\begin{equation*} \text {ACC}=\sum _{i=0}^{k} \ell (x_{i}, u_{i}). \tag{35}\end{equation*}
As expected, the ACC of “MPCWTC” is the highest because it only looks at
To verify the statements in Remark 6, we first test the equivalence of RLMPC and the MPCWTC with an accumulating prediction horizon. Since the prediction horizon of RLMPC is set to be
Trajectories of RLMPC in the learning episode under different prediction horizons (linear system).
We finally investigate the data efficiency of the proposed scheme and compare it with Q-learning [44] and PILCO [45]. The former employs the action-value function and has a very similar learning process to the proposed scheme, except that it is model-free. The latter is a model-based policy search scheme and is well known for its data efficiency. By implementing Q-learning and PILCO1 in the optimal control of (34), we observe that the amount of training data required for them to converge to the optimal policy is
B. Nonlinear System
Consider the regulation problem of a nonholonomic vehicle system
\begin{equation*} x_{k+1}=x_{k}+g(x_{k})u_{k} \tag{36}\end{equation*}
where Remark 9:
According to Brockett’s theory [46], there does not exist smooth and time-invariant feedback to stabilize system (36), which makes the widely adopted design method for terminal conditions, such as [10], no longer applicable to this system, thus posing a great challenge to the traditional MPC design here. Even if qualified terminal conditions can be found, they tend to be quite conservative, as evidenced by a small terminal set, a largely required prediction horizon, and an overestimated terminal cost. This can lead to the conservative performance of the designed traditional MPC.
1) Design of the Traditional MPC and RLMPC:
In [47], a traditional MPC is proposed for (36): the terminal cost is
Remark 10:
This traditional MPC design is computationally demanding because the prediction horizon should be at least 30 to meet the terminal constraint, which confirms Remark 9. Moreover, there is no guarantee that the control performance of the designed MPC is optimal. In contrast, RLMPC avoids artificially designing the terminal conditions and achieves (near-)optimal performance. With the learned (near-)optimal terminal cost, its prediction horizon can be flexibly adjusted without affecting its performance. Therefore, it is expected to show unique superiority in this problem.
Now we present the RLMPC design:
2) Comparison of the Traditional MPC and RLMPC:
In the following simulations, the prediction horizons of the traditional MPC and RLMPC are set to 30 and 5 steps, respectively.
As we can see from Figs. 6 and 7 that no constraint is violated during both episodes, which demonstrates that RLMPC is able to restrict the policy within the input and state constraints and thus perform safely. Fig. 8 illustrates the convergence of the weight
System trajectories and ACCs of RLMPC policies in the 1st–25th iterations of the learning episode (nonlinear system).
3) Prediction Horizon and Computational Burden:
We now demonstrate the relationship between prediction horizon, control performance, convergence, and computational burden and use the following three metrics in the quantitative evaluation: ACC, CR, and the average computational time (ACT) at each step. RLMPC in the learning episode with four different prediction horizons, namely
Consistent with the linear system example, increasing the prediction horizon of RLMPC effectively enhances CR and leads to an improved (lower ACC) trajectory in the learning episode. Conceivably, this comes at the cost of increased computation time at each step. However, we observe that even with the same prediction horizon of 30, the ACT of RLMPC is still slightly lower than that of traditional MPC. This is mainly due to the removal of the terminal constraint, which simplifies the optimization problem.
We now perform a verification of Remark 7. Taking the learned (near-)optimal value function as the terminal cost of RLMPC, we reduce the prediction horizon to 1 and re-execute the regulation task. As we can see from Fig. 10, the trajectories and input under
Comparison of trajectories and inputs of RLMPC with the learned (near-)optimal value function under prediction horizons
Conclusion
To free the design of the terminal cost, terminal controller, and terminal set in traditional MPC and to improve the closed-loop performance, this article developed a new RLMPC scheme for discrete-time systems by combining RL and MPC through PI. The core of this scheme is to take a value function, which is obtained through learning, as the terminal cost of traditional MPC to stabilize the system. We showed that the value function monotonically converges to the optimal one under PI. Based on this property, the closed-loop stability of RLMPC was further proved. We tested the proposed scheme on a linear system example to show its convergence, safety, optimality, and stability. Furthermore, we also implemented it on a nonholonomic vehicle system to show that RLMPC outperforms traditional MPC in the nonlinear case. Finally, we discussed the impact of the prediction horizon on the learning process and the control performance of RLMPC, highlighting the ability of RLMPC to flexibly adjust the prediction horizon to reduce the computational burden.
